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ABSTRACT 

The professional and personal biases of an evaluation 
team at Cornell University were found to interfere with the 
assessment of an interdisciplinary, team taught, problem oriented 
curriculum in International Studies. An account was dlven'of the 
hiring of the evaluators, evaluator role ambiguity, evaluator 
professional clashes, facilitation of group compromise, evaluation 
tasks, and models for evaluation. The personal growth that occurred 
between the evaluators as they learned together and practiced 
self-scrutiny was emphasized. Evaluation was viewed, as having a 
disciplinary, intellectual, interpersonal, and professional impact oh 
the evaluators. (BJG) 
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The purpose of this symposium is to open up for discussion a somewhat 
neglected area of professional practice. As evaluators, we work in the real 
world and must cope with high levels of uncertainty, ambiguity, and complexity. 
Recently, these problems have become more acute in situations where we shift 
roles, "become less the non-participant observer and more the program consultant 
and participant. Under these new conditions, we need to develop procedures - 
to maintain appropriate levels of professional and ethical consistency. One 
such procedure is self scrutiny, the monitoring of our own reactions and 
learnings as outcomes of the evaluation programs we design and implement. That 
is the topic of the present symposium. 



Background 

The importance of self scrutiny was particularly salient in an evalua- 
tion study that was done recently by the Center for Improvement of Undergraduate 
Education at Cornell University. As a group, we were responsible for assess- 
ing an interdisciplinary, team taught, problem oriented curriculum in Inter- 
national Studies. Because the program was interdisciplinary, complex, and 
^4 changed markedly over time, each member of the evaluation team was required 

to play a variety of roles within the evaluation design. t The result was that, 
more often than is usually the case, we quickly became sensitized to some of 
the different types of impacts that our work was having on us as professionals 
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and individuals. 

Briefly, interest in this evaluation began when the Center for In-___ 
ternational Studies organized an experimental undergraduate program. This 
program was an innovative effort "by CIS to make available to undergraduate 
students the rich, interdisciplinary resources in International Studies that 
were available at Cornell, and at the same time, to test the benefits and 
limitations of conventional introductory course teaching techniques in an 
interdisciplinary, team teaching environment. The University accepted the 
initial proposal of six different courses to be offered over a three-year 
period and also required that the program be assessed in some systematic 
fashion. 

Initially, this assessment was internal^ — a subjective review of the 
first course by the team who taught that offering. This procedure was not 
completely satisfactory to CIS and they negotiated a more formal evaluation 
with Stephen Brock-. who was then the -Associate Director of the Center for 
Improvement of Undergraduate Education. Brock, an educational educational- 
philosopher and administrator coordinated the evaluation of the next two 
courses. This effort used items from the Cornell Inventory for Student : 
Appraisal of Teaching and Courses to measure student perceptions at the end 
of the course. 

The second CIS course, on Domination^and Subordination, included a 
strong Marxist component, emphasized contemporary topics, and generated con- 
siderable controversy within the Cornell faculty and within CIS. This in 
turn led to a direct faculty challenge of CIS* right, and competence, to 



5 



- 3 - 



offer undergraduate courses. Under these new political circumstances, there 
vas a perceived need for a more comprehensive assessment of the^program. Work- 
ing with CIS, Stephen Brock negotiated support from the Institute for Interna- 
tional Studies of the United States Office of Education. This proposal de- 
lineated six major evaluation tasks. Briefly, these were: 

1. A comprehensive description of the two cburses to be 
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offered during the 1972-73 academic year; 
2. An analysis of the anticipated impacts of these courses 

on students and faculty; 
3» A similar analysis of unanticipated outcomes on students 

and faculty; 

U. Identification of shared and dissimilar objectives between 
courses; 

5* Identification of the models of interdisciplinarity used 

in these courses; A| 
6. The development of evaluation guidelines for problem 

oriented, interdisciplinary co^irses in International Studies 
To design and administer this project, Brock recruited Whiton Paine, 
an evaluation specialist.. The timing of grant approval, plus prior work 
commitments meant that major design work did not start until two weeks before 
the Fall course was scheduled to begin. Thus, severe time pressure was an 
important initial consideration. Other, unexpected considerations were that 
all members of the teaching team to be evaluated had not been informed in 
advance that their work would be monitored, and the individual primarily 
responsible for the course concluded that he had been^. inadequately consulted 



"by the CIS administration and also that the evaluation was a direct attack 
on his academic freedom. He acquiesced to only the most unobtrustive of 
evaluations. 

Major differences between Paine and Brock also developed in two im- 
portant areas. First, there was a lack of role clarity in terms of their 
areas of responsibility in the design and implementation of the evaluation 
project. This problem was complicated further' because Brock, drawing upon 
his curricular, philosophical and interdisciplinary interests, and Paine, 
responding to the specifics of the O.E. evaluation proposal and his own train 
ing in experimental psychology, had very different, and somewhat incompatible 
perceptions about the goals and appropriate methodology of the evaluation. 
In short, a typical and stressful mess at the beginning of an assessment ef 
effort. 

In this planning stage, David Macklin, a Social Psychologist was re- 
cruited to serve a variety of consulting roles. As a member of the CIUE 
evaluation group, he made a substantial contribution to the entire evaluation 
In particular, he facilitated effective communication between Brock and 
Paine, served as an observer who was not directly affected by ideological 
and interpersonal differences, and was crucial in suggesting compromises 
throughout the project. 

The result was a preliminary design whcch used a modified Context- 
Input-Process-Product (CIPP) model for a reasonably comprehensive evaluation 
of the separate courses on rural development and ethnicity. Pre-, midterm, 
and post-questionnaires, interviews with faculty, and observation of course 
activities during the semester were to be the main sources of data and 
multivariate procedures would be used to transform data into information 
-for future program decisions. This design was applied in the assessment of 
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the rural development course but a severe enrollment decline between this 
course and the next, one, led to a major reorganization of the evaluation 
effort. 

The final design added follow-up questionnaires and faculty inter- 
views for the earlier CIS courses, included an additional CIS course given 
this year r and stressed a more comprehensive consideration of the context of 
the program. This required a complete analysis of relevant material in the 
Center for International Studies files. Allison Brown, an Intellectual 
Historian' was recruited for this task because of her specialized skills in 
this area and because of her unique previous experience with the program. 

Thus four very different individuals collaborated on the evaluation 
of this interdisciplinary curriculum. Working together in ways which changed 
markedly over time, led to the identification of four main classes of impacts — 
the disciplinary, the intellectual, the interpersonal, and the professional. 
As is true for most evaluators, each member of the team had a primary dis- 
ciplinary allegiance as well as secondary, specialized disciplinary competencies. 
We also were committed to different intellectual values and styles. Both dis- ~ ~~ 
ciplinary skills and intellectual values were modified somewhat as we worked 
together as an interdisciplinary team evaluating an interdisciplinary curri- 
culum* 

Our interpersonal skills and needs also differed, and some of the impacts 
of our work are reflected in our present feelings for each other, our new 
levels of skills in this area, and in the forms of our present and expected 
future relationships. Finally, the evaluation made us master new professional 
skills and modify skills previously developed, as well as providing each of us 
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with a new insight into the evaluation enterprise an£ into the problems and 
rewards of other aspects of our professional roles. 

These impacts were specific to this evaluation hut they are seen as 
having a more general import. Every evaluation, to a greater or lesser 
degree, has disciplinary, intellectual, interpersonal, and professional im- 
pacts on evaluators* We offer this particular analysis net as an attempt to 
exhaustively delineate the characteristics of these classes of evaluation 
impacts hut rather to encourage consideration within the evaluation commu- 
nity of this neglected area of the work we do* Each of us will briefly pre- 
sent in more detail these impacts and then the floor will be opened for a \ 
more general discussion of the problem which we are attempting to delineate/ 
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CONCLUDING REMARKS 
Whiton Stewart Paine k - 

I think two main generalizations can be drawn from our Vork^on the 
task of specifying the impacts of the evaluation on ourselves. Each of us 
has, through this process, uncovered a rich store of learnings and other 
effects. The ones we saw as particularly striking were presented here today 
but there were many, more subtle disciplinary, intellectual, interpersonal, 
and professional impacts as well. 

Also, at least for us, this process of self scrutiny has 'proven to 
be very valuable in stabilizing, in identifying and clarifying important 
lessons we learned from working together. Whether or not a similar search 
has been, or will be, useful to others we do not know, and having talked 
about ourselves, now we would like to hear from you about the issues raised 
in this symposium. 
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